\chapter{Conventional Data Processing}
\label{ch:conventional_data}

\section{Overview}
\label{sec:conv_data_overview}

Conventional data processing in GSI encompasses the assimilation of traditional meteorological observations from radiosondes, surface stations, aircraft, ships, and other conventional platforms. These observations form the backbone of numerical weather prediction systems and provide crucial constraints on the atmospheric state variables including temperature, humidity, wind, and surface pressure.

The conventional data processing system is built around four primary setup routines that handle the core atmospheric variables: \texttt{setupt} for temperature, \texttt{setupw} for wind components, \texttt{setupq} for moisture, and \texttt{setupps} for surface pressure. Each setup routine implements specialized algorithms tailored to the unique characteristics and error structures of its respective observation type.

\section{Temperature Processing: \texttt{setupt}}
\label{sec:setupt}

The \texttt{setupt} subroutine handles the processing of temperature observations from various platforms, implementing sophisticated quality control, bias correction, and error specification procedures specifically designed for thermal measurements.

\subsection{Observation Sources and Characteristics}

Temperature observations processed by \texttt{setupt} include:

\begin{itemize}
    \item \textbf{Radiosonde temperature profiles}: High-vertical-resolution measurements from balloon-borne instruments providing temperature from surface to stratosphere
    \item \textbf{Aircraft temperature reports}: Commercial and research aircraft providing temperature measurements along flight paths
    \item \textbf{Surface temperature observations}: Land-based meteorological stations and automated weather stations
    \item \textbf{Marine temperature data}: Ship-based observations and fixed oceanic platforms
    \item \textbf{Profiler temperature retrievals}: Wind profiler systems with temperature retrieval capabilities
\end{itemize}

\subsection{Quality Control Procedures}

The temperature quality control system implements multi-layered screening procedures:

\subsubsection{Range Checking}
Physical plausibility checks ensure temperature observations fall within climatologically reasonable bounds:
\begin{equation}
T_{\min}(p,\phi,\lambda,t) \leq T_{obs} \leq T_{\max}(p,\phi,\lambda,t)
\end{equation}
where the bounds depend on pressure level $p$, latitude $\phi$, longitude $\lambda$, and time $t$.

\subsubsection{Background Check}
Observations are compared against background field estimates using normalized departures:
\begin{equation}
\chi^2 = \frac{(T_{obs} - T_{bg})^2}{\sigma_{obs}^2 + \sigma_{bg}^2}
\end{equation}
where $\sigma_{obs}$ and $\sigma_{bg}$ represent observation and background error standard deviations, respectively.

\subsubsection{Buddy Check}
Spatial consistency verification compares observations with nearby measurements:
\begin{equation}
\Delta T_{buddy} = |T_{obs} - \overline{T_{neighbor}}| \leq \alpha \cdot \sigma_{spatial}
\end{equation}
where $\alpha$ is a platform-dependent threshold factor.

\subsection{Bias Correction}

Temperature bias correction addresses systematic errors in different observing platforms:

\subsubsection{Aircraft Temperature Bias}
Aircraft temperature measurements require altitude-dependent bias corrections:
\begin{equation}
T_{corrected} = T_{obs} - \beta_0 - \beta_1 \cdot p - \beta_2 \cdot p^2
\end{equation}
where $\beta_0, \beta_1, \beta_2$ are empirically determined coefficients.

\subsubsection{Radiosonde Bias Correction}
Systematic biases in radiosonde temperature measurements vary by:
\begin{itemize}
    \item Instrument type and manufacturer
    \item Solar radiation effects (daytime vs. nighttime)
    \item Pressure level and atmospheric conditions
    \item Geographic location and season
\end{itemize}

\section{Wind Processing: \texttt{setupw}}
\label{sec:setupw}

The \texttt{setupw} subroutine processes horizontal wind component observations, handling the vector nature of wind measurements and their associated coordinate transformations and error correlations.

\subsection{Wind Vector Representation}

Wind observations are processed as horizontal vector components:
\begin{equation}
\mathbf{V} = u\mathbf{i} + v\mathbf{j}
\end{equation}
where $u$ and $v$ represent the zonal and meridional wind components, respectively.

\subsection{Coordinate System Transformations}

Wind observations from different platforms require coordinate system transformations:

\subsubsection{Earth-Relative to Grid-Relative}
Transformation from earth-relative coordinates to model grid coordinates:
\begin{equation}
\begin{pmatrix} u_{grid} \\ v_{grid} \end{pmatrix} = \begin{pmatrix} \cos\alpha & \sin\alpha \\ -\sin\alpha & \cos\alpha \end{pmatrix} \begin{pmatrix} u_{earth} \\ v_{earth} \end{pmatrix}
\end{equation}
where $\alpha$ is the grid rotation angle.

\subsubsection{Platform-Specific Transformations}
Different observing platforms provide wind measurements in various reference frames:
\begin{itemize}
    \item Aircraft: True airspeed corrections for platform motion
    \item Radiosondes: Balloon drift corrections and GPS positioning
    \item Satellite winds: Geometric and temporal collocation adjustments
    \item Surface stations: Anemometer height adjustments and exposure corrections
\end{itemize}

\subsection{Wind Error Modeling}

Wind observation errors exhibit complex covariance structures:

\subsubsection{Error Covariance Matrix}
The wind error covariance matrix accounts for correlations between $u$ and $v$ components:
\begin{equation}
\mathbf{R}_{wind} = \begin{pmatrix} \sigma_u^2 & \sigma_{uv} \\ \sigma_{uv} & \sigma_v^2 \end{pmatrix}
\end{equation}

\subsubsection{Velocity-Dependent Error Scaling}
Error magnitudes often scale with wind speed:
\begin{equation}
\sigma_{wind}(V) = \sigma_0 + \sigma_1 \cdot V + \sigma_2 \cdot V^2
\end{equation}
where $V = \sqrt{u^2 + v^2}$ is the wind speed magnitude.

\section{Moisture Processing: \texttt{setupq}}
\label{sec:setupq}

The \texttt{setupq} subroutine handles the processing of atmospheric moisture observations, including specific humidity, relative humidity, and dewpoint temperature measurements.

\subsection{Moisture Variable Conversions}

Different platforms provide moisture information in various formats requiring standardization:

\subsubsection{Specific Humidity}
The primary moisture variable in GSI:
\begin{equation}
q = \frac{m_v}{m_{dry} + m_v}
\end{equation}
where $m_v$ is water vapor mass and $m_{dry}$ is dry air mass.

\subsubsection{Relative Humidity Conversion}
Converting relative humidity observations to specific humidity:
\begin{equation}
q = \frac{0.622 \cdot RH \cdot e_s(T)}{p - RH \cdot e_s(T)}
\end{equation}
where $e_s(T)$ is the saturation vapor pressure at temperature $T$.

\subsubsection{Dewpoint Temperature Conversion}
Converting dewpoint observations:
\begin{equation}
q = \frac{0.622 \cdot e_s(T_d)}{p - e_s(T_d)}
\end{equation}
where $T_d$ is the dewpoint temperature.

\subsection{Moisture Quality Control}

Moisture observations require specialized quality control procedures:

\subsubsection{Supersaturation Checks}
Ensuring physical consistency with respect to saturation:
\begin{equation}
RH = \frac{e}{e_s(T)} \leq RH_{max}
\end{equation}
where $RH_{max}$ is typically set to 1.05 to account for measurement uncertainties.

\subsubsection{Vertical Consistency}
Moisture profiles must exhibit physically realistic vertical structures:
\begin{itemize}
    \item Decreasing mixing ratios with altitude in the troposphere
    \item Consistency with temperature lapse rates
    \item Reasonable boundary layer moisture gradients
\end{itemize}

\subsection{Moisture Error Characteristics}

Moisture observation errors exhibit strong dependencies on:

\subsubsection{Relative Humidity Dependence}
Error characteristics vary dramatically with moisture content:
\begin{equation}
\sigma_q = \sigma_{dry} + (\sigma_{moist} - \sigma_{dry}) \cdot \frac{RH}{100}
\end{equation}

\subsubsection{Altitude Dependence}
Moisture errors increase significantly with altitude due to decreasing absolute moisture content and instrument sensitivity limitations.

\section{Surface Pressure Processing: \texttt{setupps}}
\label{sec:setupps}

The \texttt{setupps} subroutine processes surface pressure observations, implementing sophisticated algorithms for handling station elevation corrections and temporal variations.

\subsection{Station Elevation Corrections}

Surface pressure observations require reduction to a common reference level:

\subsubsection{Standard Pressure Reduction}
Converting station pressure to sea level pressure:
\begin{equation}
p_{SL} = p_{station} \cdot \exp\left(\frac{g \cdot h}{R \cdot T_v}\right)
\end{equation}
where $h$ is station elevation, $g$ is gravitational acceleration, $R$ is the gas constant, and $T_v$ is virtual temperature.

\subsubsection{Model Orography Interpolation}
Adjusting observations to model surface elevation:
\begin{equation}
p_{model} = p_{obs} \cdot \left(\frac{p_{surface,model}}{p_{surface,obs}}\right)
\end{equation}

\subsection{Temporal Processing}

Surface pressure observations exhibit significant temporal variations requiring careful processing:

\subsubsection{Diurnal Cycle Correction}
Accounting for semi-diurnal pressure oscillations:
\begin{equation}
p_{corrected} = p_{obs} - A \cdot \cos(2\omega t + \phi)
\end{equation}
where $A$ is the amplitude, $\omega$ is the angular frequency, and $\phi$ is the phase.

\subsubsection{Trend Analysis}
Identifying and correcting for synoptic-scale pressure tendencies:
\begin{equation}
\frac{\partial p}{\partial t} = \alpha_1 \cdot (p_{t+1} - p_{t-1}) + \alpha_2 \cdot (p_{t+2} - p_{t-2})
\end{equation}

\section{Configuration System: \texttt{convinfo}}
\label{sec:convinfo}

The \texttt{convinfo} configuration system provides centralized control over conventional data processing parameters through a structured text file format that defines observation types, usage flags, and processing parameters.

\subsection{File Structure and Format}

The \texttt{convinfo} file consists of records with the following structure:
\begin{verbatim}
obstype  platform  instrument  satellite  usage  twindow  numqc  qcparams  error
\end{verbatim}

\subsubsection{Observation Type Codes}
Standardized codes identify different observation types:
\begin{itemize}
    \item 120: Radiosonde temperature
    \item 220: Radiosonde relative humidity  
    \item 242: Radiosonde wind
    \item 181: Surface pressure
    \item 187: Surface temperature
\end{itemize}

\subsubsection{Usage Flags}
Control assimilation behavior:
\begin{itemize}
    \item 1: Assimilate observation
    \item 0: Monitor only (no assimilation impact)
    \item -1: Reject observation
\end{itemize}

\subsection{Quality Control Configuration}

\subsubsection{Thinning Parameters}
Spatial and temporal thinning controls:
\begin{itemize}
    \item Horizontal thinning distance (km)
    \item Vertical thinning layers
    \item Temporal thinning window (hours)
\end{itemize}

\subsubsection{Error Specification}
Observation error variances and correlations:
\begin{equation}
\sigma_{obs}^2 = \sigma_{base}^2 \cdot (1 + \alpha \cdot f_{error\_inflation})
\end{equation}

\section{Bias Correction Framework}
\label{sec:bias_correction}

The conventional data bias correction system addresses systematic errors in observational data through adaptive algorithms that estimate and remove platform-specific biases.

\subsection{Adaptive Bias Correction}

Time-varying bias estimation using recursive algorithms:
\begin{equation}
\beta_{k+1} = \beta_k + K_k \cdot (d_k - H_k \cdot x_k)
\end{equation}
where $\beta$ represents bias coefficients, $K$ is the Kalman gain, and $d$ is the observation-minus-forecast departure.

\subsection{Cross-Platform Calibration}

Inter-platform bias relationships:
\begin{itemize}
    \item Reference platform designation
    \item Relative bias coefficient estimation
    \item Temporal stability monitoring
    \item Geographic dependency analysis
\end{itemize}

\section{Performance Optimization}
\label{sec:conv_performance}

\subsection{Computational Efficiency}
Optimization strategies for large observation volumes:
\begin{itemize}
    \item Vectorized quality control algorithms
    \item Efficient spatial searching algorithms
    \item Memory-optimized data structures
    \item Load balancing across processors
\end{itemize}

\subsection{Parallel Processing}
Effective parallelization approaches:
\begin{itemize}
    \item Observation-level parallelism
    \item Domain decomposition strategies
    \item Communication minimization
    \item Load balancing algorithms
\end{itemize}

The conventional data processing system provides the foundation for assimilating traditional meteorological observations in GSI, ensuring that these critical measurements are optimally utilized in the data assimilation process through sophisticated quality control, bias correction, and error specification procedures.